COMMENT ⊗   VALID 00007 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.SELECT A
C00014 00003	.SELECT C
C00023 00004	.GROUP SKIP 2
C00027 00005	.NEXT PAGE
C00032 00006	.NEXT PAGE
C00040 00007	.GROUP SKIP 2
C00043 ENDMK
C⊗;
.SELECT A
2. FREQUENCY MODULATION SYNTHESIS
.GROUP SKIP 1
.SELECT C
INTRODUCTION TO SYNTHESIS AND ANALYSIS TECHNIQUES
.SELECT 1
.BEGIN FILL ADJUST
The second method of synthesis is by means of frequency modulation (FM).
Our research using this technique has yielded some altogether surprising
results, in that a simple algorithm is capable of producing a large number
of highly differentiated tones which can have a strong perceptual
resemblance to those of natural musical instruments.  The technique is
based on the sinusoidal modulation of a carrier wave, where both the
modulating and carrier frequencies are in the audio band.  When the carrier
and modulating frequencies are small integer multiples of some frequency,
the components of the modulated carrier are in a harmonic sequence and
when they are related by irrational multiples, they are inharmonic.  Thus,
by assigning the ratio of the carrier to modulating frequencies, a large
number of harmonic and inharmonic spectra can be produced.  The number of
significant components in the spectrum is determined by the frequency deviation
or amplitude of the modulating wave.  Thus, as the deviation increases from zero,
more and more components become significant, or we say the bandwidth of the
signal increases.  In fact, the bandwidth relates indirectly to the deviation
and directly to the modulation index which is the ratio of the deviation to
the modulating frequency.  The power of this technique for synthesis is that
the relationship of frequencies and the bandwidth of the signal can be controlled
by two parameters.  When the modulation index changes as a function of time,
the bandwidth of the signal changes more or less proportionally, as shown in Figure 7.
.END
.NEXT PAGE
.SELECT 5
synthesis
.SELECT 1
.BEGIN FILL ADJUST
Here we produce  the tone by varying  the amplitudes and phases  of a
small   number  of  sinusoids.     The  feature  which  distinguishes
modulation synthesis from additive synthesis  is that one or more  of
the parameters  is no longer restricted  to be a slowly  time varying
function.  A more  complete description of  FM synthesis  is given in
Chowning (1973),included as Appendix E.  We shall describe briefly the
essence  of the method. The basic equation is shown in equation (2).
.END

%3(2)     F%8α%3 = A sin(%4w%8c%3αh + I sin(%4w%8m%3αh))

Notation:    F%8α%3 is the sampled, digitized waveform at time αh
	     A is the amplitude of the modulated carrier
	     %4w%8c%3 is 2π times the carrier frequency
	     %4w%8m%3 is 2π times the modulating frequency
	     I is the peak frequency deviation/modulating frequency
	     h is the time between consecutive samples
.SELECT 1
.BEGIN FILL ADJUST
For the basic application, we restrict the modulating
waveform to a pure sinusoid. `%3I%1' is called the `modulation index,' which is
the ratio of the frequency deviation to the modulating frequency.  For
most of the useful applications we allow the amplitude A and the modulation index
'%3I%1' to vary slowly with time, while %4w%8c%1 and %4w%8m%1 are constant with
time.
.END
	We may expand equation (2) as follows:
.SELECT 3

(3)     F%8α%3 = A{J%80%3(I) sin(%4w%8c%3αh)
		  + J%81%3(I)[sin((%4w%8c%3+%4w%8m%3)αh)-sin((%4w%8c%3-%4w%8m%3)αh)]
		  + J%82%3(I)[sin((%4w%8c%3+2%4w%8m%3)αh)+sin((%4w%8c%3-2%4w%8m%3)αh)]
		  + J%83%3(I)[sin((%4w%8c%3+3%4w%8m%3)αh)-sin((%4w%8c%3-3%4w%8m%3)αh)]
		  + . . . }
.SELECT 1
.BEGIN FILL ADJUST
where J%8i%1(I) is the ith Bessel function of the first kind as a
function of the modulating index, I. Thus we see that the tone
represented by the waveform of equation (2) consists of a series of
sinusoidal componants whose frequencies are determined by the sum of the
carrier frequency, %4w%8c%1, and integral (positive and negative) multiples
of the modulating frequency, %4w%8m%1. As the modulation index increases,
the amplitudes of the components change in complex ways, but the general trend
is that energy is shifted away from the carrier frequency. When the modulation
index is zero, equations (2) and (3) degenerate to a sinusoid at
the carrier frequency. As the modulation index increases, more energy is
transferred to components representing larger and larger integral multiples of the
.NEXT PAGE
modulation frequency, as shown in Figure 1, Appendix E.

If the carrier frequency and the modulating frequency are small integer
multiples of some other frequency, %4w%1, then the components form a harmonic
sequence. If the carrier and modulating frequencies are related by
irrational multiples, then the components are inharmonic.

Both of these cases have use in synthesis and both are dependent upon
reflected lower side-frequencies where all or some of the components
produced by the terms %4w%8c%1-k%4w%8m%1 are less than zero.
Since this is equivalent to a positive frequency with
a phase shift, there are contributions which might
not be obvious at first glance. For example, in the case %4w%8c%1 = %4w%8m%1,
the components in the negative frequency domain reflect around 0 Hz
and add to the components in the positive domain thus forming the harmonic
series where the partial at %4w%8c%1 will have amplitude (J%80%1(I)-J%82%1(I)),
the partial at 2%4w%8c%1 will have amplitude (J%81%1(I)+J%83%1(I)), Figure 4,
Appendix E.  For the case where %4w%8m%1 = 2%4w%8c%1, the only nonzero components will be at
odd multiples of %4w%8c%1, with amplitudes again being sums of pairs
of bessel functions.  %4w%8c%1 does not always represent the fundamental
frequency of the tone. For instance, if %4w%8c%1 = 2%4w%8m%1, then a harmonic
series based on %4w%8m%1 is produced.  If, on the other hand,
%4w%8m%1 = 2%21/2%4w%8c%1, the reflected components will inter-leave with
the positive components producing an inharmonic spectrum, Figure 6, Appendix E.

It should be pointed out that while the power of this technique of producing
periodic signals is very great, %5it does require a degree of precision that
can only be attained by digital synthesis techniques,%1 viz. if, in the case
of a ratio of %4w%8c%1/%4w%8m%1 = 1/1, there were frequency drift of 1Hz in either
oscillator, the reflected lower side frequencies and the upper side frequencies
would have a 2Hz difference producing a clearly audible beat.  Although
it is an effect that can be useful, it must certainly be under control.
.END


.SELECT 5
FM predictive analysis and graphic techniques
.SELECT 1
.BEGIN FILL ADJUST
At the present stage of research, we rely upon the
intuition of the researcher to develop suitable parameter values and
amplitude and index functions. There are tools we have given the researcher,
however, to aid in exploring the consequences of the various choices.
The most powerful of these tools is an interactive graphical program which
allows the user to design amplitude and index functions for evaluating
equation (3). The program makes a perspective plot of the amplitudes
of the partials as a function of time. Figure 7 shows such a plot where
the amplitude is constant, the modulation index increases in time from
0 to 8, and the ratio of %4w%8c%1/%4w%8m%1 = 1.  After the display is produced,
the user can then go back and alter his function shapes and parameters
to try to converge on the desired spectral characteristics.
.END
.GROUP SKIP 2
.SELECT C
CURRENT RESEARCH
.SELECT 1
.BEGIN FILL ADJUST
Frequency modulation synthesis, unlike additive synthesis, is not at
the outset an obvious model for tones which have any likeness to those of natural
instruments.  The process by which these applications were discovered was
initially founded in the ability of a trained `musical ear' to penetrate
the spectral complexity of the FM technique and to predict the effect of
a change to the parameters of the equation.  The development of analysis
and synthesis programs, as described in the previous section, yields precise
data describing a complex signal which significantly increases our ability
to apply the FM technique to the simulation of music-instrument tones.

We establish in this section the correlation of the parameters of the FM
equation to perceptual cues.  We select ratios of the carrier to modulating
frequencies that produce harmonic spectra appropriate to the simulation.
For example, 1/1 for brass tones, which includes all of the harmonics and
1/2 for clarinet tones, which includes only the odd numbered harmonics.
A clarinet-like tone is presented on Recorded Example 2.

The critical correlation for the production of natural sounding tones
is the change of the modulation index as a function of time to the evolution
of the bandwidth of the resulting spectrum.  Based on the analysis of
brass tones by Risset (1966), we were able to synthesize
natural sounding brass tones by simply relating the evolution of the
bandwidth to the amplitude envelope, as shown in Figure 8, and on
Recorded Example 2.  The initial success of this simulation was a surprise
in that no one could have predicted on the basis of psychoacoustical
knowledge the possibility of such a simple representation of a brass tone.

The FM technique can be extended by adding another carrier wave to the
system.  If the ratio of the first carrier to the modulating frequency is
1/1 and the second carrier is nine times the first, but modulated by
the same modulating wave, then additional energy is added in the region of
the ninth harmonic which is similar to a resonance.  Figures 9 and 10 show
the spectral distribution resulting  from the two modulated
carrier waves.  The bandwidth and amplitude of the resonance can be
controlled independently from the the components produced by the first
carrier, giving great additional flexibility to the technique.
.END
.GROUP SKIP 2
.SELECT 5
periodicity - ratio of carrier to modulating frequencies
.SELECT 1
.BEGIN FILL ADJUST
Since music instrument waveforms are for the most part quasi-periodic,
where successive periods of the wave have only small deviations, we
select ratios for the carrier and modulating frequencies
which produce side frequencies that fall in the harmonic series.
As we pointed out in the section on synthesis technique, small integer
multiples of some frequency will produce periodic waveforms.  Within
this generalization there are a number of further options: %4w%8c%1/%4w%8m%1 = 1,
spectrum including all of the partials; %4w%8c%1/%4w%8m%1 = 1/2, spectrum composed of
odd numbered partials; %4w%8c%1/%4w%8m%1 = 1/3, spectrum where every third partial is
missing, and so on.  
.END
.GROUP SKIP 2
.SELECT 5
bandwidth as a function of time - modulation index
.SELECT 1
.BEGIN FILL ADJUST
An initial period of experimentation resulted in convincing
simulations of several music-instrument tones of which the most surprising
were those of brass-instrument.  These tones were analyzed
in order to determine the perceptual correlations to the parameters of the
equation (2).  The equation is restated where A and I are functions of time.
.END
.SELECT 3

(4)     F%8α%3 = A%8α%3 sin(%4w%8c%3αh + I%8α%3 sin(%4w%8m%3αh))

.SELECT 1
.BEGIN FILL ADJUST
The amplitude A and the modulating
index `%3I%1' were selected to be slowly varying functions of time.
The function A%8α%1 controls the time-varying amplitude of the frequency
modulated carrier to produce the appropriate attack and
decay characteristics of the tone. The function %3I%8α%1
is used to control the evolution of the bandwidth of the spectrum.  Since in
brass tones the energy of the higher partials increases roughly in proportion
to the increase in intensity during the attack (Risset, 1966), the shape of
the amplitude function was used for the index function in order to produce
this correspondence.  The value of the index was set to range from a minimum
of 0 to a maximum of 5 according to the function shape, which produces
nine significant partials at the peak amplitude of the wave.
Figure 8 is a perspective plot of the spectrum of an FM synthesized
brass tone with the functions controlling A and %3I%1 at the top.
The increase of bandwidth with amplitude can easily be seen.

The extent to which the application of this technique is able to simulate the
natural qualities of a brass-tone is remarkable and serves to emphasize an
important physical correlate to perception: %5the temporal evolution of
the bandwidth of a signal%1.

Our  preliminary investigations  suggest that  the  evolution of  the
bandwidth  is  of  varying degrees  of  perceptual  importance, often
primary, in  all  instrument  tones  and a  large  number  have  been
synthesized using equation (3),  where the only time-domain functions
were amplitude and modulation index (Chowning, 1973).
.END
.GROUP SKIP 2
.SELECT 5
resonances - multiple carrier waves
.SELECT 1
.BEGIN FILL ADJUST
There are many signals which have spectra for which there is no
approximation using the FM equation in its simple form; in
particular, the spectrum may have two regions of significant energy 
which are caused by the natural resonances of the instrument.  There
may be no value of the modulation index applied to equation (3),
which will produce a spectral curve that preserves the effect of these resonances.  
In this case, we extend the FM synthesis algorithm to include additional
carrier waves whose frequencies are set at the harmonics around which the
regions of spectral energy are centered.  The expanded equations (5,6)
include a second carrier wave.  For the purpose of simplicity in the notation,
A and I are assumed to be slowly varying functions of time as explicitly
notated in equation (3).
.END
.SELECT 3

(5)	M = I sin(%4w%8m%3αh)
(6)     F%8α%3 = A sin(%4w%8c%71%3αh + M) + K%81%3A sin(%4w%8c%72%3αh + K%82%3M)

Notation:   %4w%8c%71%3 is the first carrier frequency
	    %4w%8c%72%3 is the second carrier frequency
	    K%81%3 is a constant to scale the amplitude function for the
	           the second carrier
	    K%82%3 is a constant to scale the index function for the
	           second carrier
.SELECT 1
.BEGIN FILL ADJUST
An application of this technique is appropriate in simulating the spectral
distribution of a string tone which has a resonance near the ninth harmonic.
We break the desired spectrum into two parts, each
of which outlines a spectral shape which seems similar to an FM produced
spectral shape.  In this case the ratio of the first carrier to the modulating frequency
is set to 1, and the index ranges from a maximum of 4, during the attack,
to 1.5, where the amplitude is maximum.
This produces a spectrum shown in Figure 9.
The second carrier has a ratio to the modulating frequency of 9/1, the index
is scaled by the constant .5, to produce the index range 2 to .5,
and the amplitude is scaled down by the constant
0.3.  The spectrum of this part is shown in Figure 10.  The sum of these
two spectra has attributes of a string tone.
.END
.NEXT PAGE
.SELECT C
PROPOSED RESEARCH
.SELECT 1
.BEGIN FILL ADJUST
The research that we have described above has been heavily dependent
upon the interaction in application of the two synthesis techniques to
identical or similar signals.  Each technique has provided insights into
the critical cues which have been applied to the other as a means of
confirmation.  Although the number of signals we have considered is a
small fraction of the total we need in order to generalize the synthesis
procedures and ultimately theorize a timbre space, the number does include
highly diversified spectra which have %5given in%1 to our procedures of
reduction and FM synthesis.  We are therefore confident that within the
general constraint of quasi-periodicity we can successfully apply these
techniques to a large number of signals.

In order to pursue further the FM synthesis of some music-instrument tones,
we must first extend the technique of the synthesis of resonances.  Here,
we rely upon the results of the analysis, data reduction, and additive synthesis
to quantify the fixed resonances of instrument tones such as the violin.
On the basis of this information we will construct a resonance table for
the instrument.  The table will contain
three values for each of the tempered pitches in the normal range of the
instrument describing, 1) the harmonic around which the harmonic occurs,
2) the relative amplitude of the peak of the resonance, and 3) the bandwidth
of the resonance.  Figure is an example of such a resonance where the
values are harmonic = 7, amplitude = .3, and bandwidth = ca 1 at the
peak amplitude of the tone (as noted before, naturalness is dependent on
significant change in the bandwidth, especially during the attack).

A very powerful means of simulating secondary features of tones such as
inharmonicity during the attack period, is the use of a %5complex%1
modulating wave.  In the case of string tones the noise or scratch
is of very great importance.  In Recorded Example 2, we produce such a tone
by making the ratio of the frequency of a second modulating wave irrational
with respect to the carrier frequency, and the modulation non-zero only
during the attack.  Of particular importance in relation to the additive
synthesis technique where the attack noise resulted from asynchronous
frequency perturbation of each of the harmonics, is the apparent latitude
in producing this cue.  We will investigate, therefore, the application of
complex modulating waves to a variety of other instrument tones which have
significant noise components.

To extend the timbral range of simulation, we will also apply the technique to
the synthesis of non-periodic signals in order to measure the
correspondence of critical cues such as bandwidth in the two classes of signals.
The FM technique has very great potential in this class of timbres as demonstrated
in the Recorded Example 2.
.END
.NEXT PAGE
.SELECT 5
simulation of fixed resonances
.SELECT 1
.BEGIN FILL ADJUST
Our research in the simulation of music-instrument tones, using the additive
and FM synthesis techniques, has focused on single representative tones.
In order to extend and test the data reduction procedures we must apply
them to the analyzed data from a music-instrument within a range of
frequencies and articulations.  It is well-known that many instruments
have fixed resonances and that these resonances cause significant changes
to the shape of the spectrum as the frequency of the fundamental changes.
The analysis-synthesis-reduction techniques should highlight these effects
and allow us to produce spectral envelopes in a concise form for the
entire frequency range of an instrument.

In order to account for the effect of the fixed resonance we must provide
for a higher level of control.  The obvious analog to the resonances of a
musical instrument are band-pass filters or resonators which
are tuned to the observed resonances of the instrument in question.
As the period of the signal changes, the harmonics shift their position
in relation to the resonators thereby effecting the desirable spectral
shaping.

Mathews (1973) has shown that a large number of resonances
are exhibited by stringed instruments.  If we were to digitally
simulate the physical waveform of the violin, based on results of
research in the physics of stringed instrument tone production,
the computations required would be extremely extensive.
One realization of resonances that is suggested by the FM technique is
the form shown in equations (5) and (6) which has two carrier waves.
Two carrier waves can synthesize one (or to some extent, two) fixed resonances.

The additional carrier wave provides for arbitrary placement, amplitude,
and bandwidth of a resonance for any particular fundamental pitch period.
The second carrier frequency would be set at the partial number which is
closest to the center of the resonance for a given fundamental frequency, and the
bandwidth and peak amplitude of the resonance would be controlled by the
appropriate scale factors.  It should be pointed out that in typical
applications, the second carrier frequency may be 5 to 10 times the modulating
frequency and the index relatively small.  Since in such a case there are
no significant %5reflected%1 side frequencies, the components are symmetrical
about the carrier and have spectral envelopes which are similar to those of
natural resonances.  The three controlling values would represent the best
approximation of the spectral envelope as determined by the
analysis-synthesis-reduction technique and as constrained by the
form of equations (5) and (6).  This procedure will be repeated
for the n pitch periods of interest and a 3 by n table generated to store
the values for harmonic center, amplitude scale, and index scale.
The FM synthesis algorithm would simply extract the values according to the
scale step and synthesize the tone accordingly.  Figure 11 is a
spectral representation of a set of values applied to the
second carrier wave of equations (5) and (6).
In this manner, we may synthesize with little additional computation the
effect of a single fixed spectral resonance peak with a frequency-modulation
instrument having two carriers.
.END
.GROUP SKIP 2
.SELECT 5
inharmonicity - multiple modulating waves
.SELECT 1
.BEGIN FILL ADJUST
One of the secondary characteristics of many tones, and in particular
string tones, is the noise or `scratch' which
is apparent during the attack.  The heterodyne analysis and additive synthesis
techniques indicate and confirm respectively, that the noise is a result of
frequency disturbance of the harmonics during the attack, shown in Figure
4.  The reduction techniques have demonstrated that nearly any inharmonicity
during the attack will preserve the characteristic `scratch.'  In order to
simulate this using the FM technique, we introduce an additional modulating
wave which has an irrational ratio to the fundamental and which has non-zero
index only during the attack portion of the tone.
Equation (7) shows two sinusoidal modulating waves and a single carrier wave.
.END
.SELECT 3

(7)   F%8α%3 = A sin(%4w%8c%3αh + I%81%3 sin(%4w%8m%71%3αh) + I%82%3 sin(%4w%8m%72%3αh))
.SELECT 1
.BEGIN FILL ADJUST
In this case where the modulating function is not a single
sinusoid, but a sum of sinusoids, the resulting expansion is similar
to equation (3); however, the components  will be at the set of frequencies
consisting of the %5sum%1 of all integral multiples of the modulating
frequencies added to the carrier frequency. The amplitudes of the
components will be products of Bessel functions of the modulation
indices. We can produce combinations of harmonic and inharmonic spectra
simply by the choice of the modulating frequencies.

The simulation of a violin tone would thus far incorporate, then, both two carrier
and two modulating waves for the closest approximation.
.END
.GROUP SKIP 2
.SELECT 5
non-periodic tones
.SELECT 1
.BEGIN FILL ADJUST
We have already demonstrated in the laboratory the utility of
the FM technique for simulating non-periodic music-instrument tones
such as bells, drums, gongs, etc.  This class of tones has the
common characteristics of primary 
frequency components which do not fall
in the harmonic series, decay times which are long compared to the
attack and steady state (if any), and decrease in bandwidth which
follows, more or less, the decay.  In FM synthesis
the frequency distribution of such a tone
is controlled by setting the ratio of the carrier and
modulating frequencies to be irrational, and the bandwidth is controlled
by allowing the modulation index to follow the decay curve.  In most
cases the spectrum degenerates to a sinusoidal oscillation as the amplitude
of the tone goes to zero, in which case the index also goes to zero.
The spectral evolution of a bell tone is shown in Figure 12, and on
Recorded Example 2.

The spectral evolution in time of non-periodic tones has features
which are common also to periodic tones.  The dimensions and
structure of the timbre space must account for both classes of
tones.  As an example, we have determined through the FM technique,
that the only difference between the tone of a plucked string
and a bell is one of periodicity, or a wave composed of harmonically
related frequencies, and non-periodicity, the
amplitude envelope and evolution of the bandwidth being common.
We can imagine, then, a spatial representation of timbre where the
variance for this case is along a single dimension.
.END
.GROUP SKIP 2